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School Climate, Population Characteristics, and Program Quality 
Denise C. Gottfredson, University of Maryland 

The 1990's have been a period of great optimism about prevention. Several recent 
reviews of the prevention literature have concluded that at least some forms of 
prevention are “effective” in reducing delinquency and substance use. Federal 
agencies are busy disseminating pamphlets and various how to guides on “proven” 
prevention strategies. 

For a number of years, Gary Gottfredson and I have been interested in what 
happens when these research-based strategies are attempted outside of the context of 
the original research. In one three-year study of a discipline management intervention 
(Gottfredson, Gottfredson, and Hybl, 1993), we documented considerable variability in 
the quality of implementation across the six participating schools, and found that the 
size of the positive effects on student behavior outcomes was a function of the quality 
of implementation. In a subsequent demonstration project involving only one troubled 
urban middle school (Gottfredson, Gottfredson, and Skroban, 1998), we attempted to 
implement the very same prevention strategies that had been highlighted in research 
reviews as especially effective for reducing substance use and violence. We 
incorporated several of these programs into a set of courses to be delivered by regular 
school teachers to all students in the school. After a four-year struggle to get these in 
place, we concluded that the program was never fully implemented, and as 
implemented it had no discernable effects on substance use, any other form of problem 
behavior, or even any of the several predictors of problem behavior we measured. 

Our observations during these prevention projects and several others only served 
to increase our interest in understanding factors related to variability in implementation. 
We began to suspect that the positive findings from research studies might not 
generalize to real-world settings, at least not the type of places we had been selecting 
for our work. I think we also wanted to assure ourselves that it was not just everything 
we did that failed - that weak implementation was a more general problem. As we dug 
deeper, we began to find snippets of evidence to support the belief that prevention 
research results do not seem to be highly generalize-able to natural settings. 

We saw, for example, in Lipsey’s (1992) meta-analyses of prevention and treatment 
programs that programs run by researchers have larger effects than programs not run 
by researchers. And evidence from Nan Tobler’s (1992) meta-analysis of evaluations 
of drug prevention programs showed that the most effective programs were run by 
somebody other than the regular classroom teachers. 

We found some useful work by Gil Botvin and his colleagues (Botvin, Baker, 
Dusenbury, Tortu and Botvin, 1990) that demonstrated huge amounts of variability in 
the implementation of his substance abuse prevention program, Life Skills Training. 

His team carefully measured the amount of the LST curriculum delivered after teacher 
training. The percentage of the materials covered in actual implementation varied 
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widely from school to school - from 27% to 97%, with an average of 68%. Only three- 
fourths of the students were exposed to at least 60% of the program. Botvin also 
showed that when the program is delivered incompletely, positive effects are not found. 
In fact, he typically excludes from his outcome studies those classrooms which 
delivered less than 60% of the program. 

All of this pointed to the same tentative conclusion - that the strength and integrity 
of program implementation matters, and that the results of research studies of 
prevention may not generalize to everyday situations. But it was clear that we were 
only looking at a few pieces of a much larger puzzle, and that more research into the 
issue of implementation was needed. 

As a first step in developing the instruments for the study, we collected information 
about existing programs from practitioner handbooks and other publications, research 
articles, funding agency records, and so on and developed a classification of prevention 
services that are provided by or in schools. The long list of activities discovered through 
this process is shown in Figure 1 . We wanted to be sure to ask schools about all of 
these different activities so that we could describe the entire range of possible 
prevention activities. Next, based on existing literature on variability in implementation 
(mostly from studies of educational reform efforts), we developed hypotheses about 
factors leading to successful program implementation. These are shown in Figure 2. 
Our over-arching hypothesis is that we would observe a “Matthew Effect,” that is, that 
the schools with the most challenging problems would benefit the least from prevention 
programs because the quality of implementation would suffer in these schools. 

We learned in Phase 1 of the study (during which we asked principals to report 
about the number and types of prevention activities in place in their schools) that the 
typical school has 14 different unique prevention activities going on at once! Actually, 
this understates the number of prevention activities because it includes only the 
fourteen categories of discretionary prevention activities we asked about. That is, it 
excludes such activities as school-wide discipline, which all schools have, and 
architectural innovations.lt is also a low estimate because we included on the 
questionnaire booklet spaces for only five different programs of each type. Each of the 
14 different types of prevention we asked about were claimed frequently, with 
prevention curricula being the most popular - used in 76% of schools, and “Youth roles 
in regulating and responding to conduct” - mostly peer mediation programs and student 
courts - used in 40% of the schools. This suggests that we need more research on a 
wider range of school-based prevention strategies. The wide variety of programs 
currently in place in schools contrasts sharply with the relatively narrow range of 
strategies that has been studied rigorously. We know little or nothing about the 
effectiveness of most of the strategies currently in use in schools. 

Is all of this prevention activity useful? We expect that it depends upon the quality 
of implementation of each activity. If the consequence of having more activities is to 
implement each one less rigorously, it is probably not useful to have so many different 
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activities. The remainder of my talk addresses the quality of implementation of these 
prevention activities. 



The data on program quality for the first fourteen categories of prevention come 
from the activity questionnaires. We developed a tailored questionnaire for each 
prevention activity to measure each of the hypothesized factors leading to high 
implementation as well as several different indicators of the quality of implementation. 

It was necessary to tailor these surveys because indicators of quality are different 
across different types of prevention. For example, a high-quality behavior modification 
program would look very different from a high quality security and surveillance program. 



We randomly sampled within school from the over 17,000 prevention activities 
named by principals in Phase 1 . We sampled one activity from each of the first 
fourteen categories of prevention and selected with certainty all D.A.R.E. and peer 
mediation programs for closer scrutiny. This resulted in a sample of just over 8,000 
prevention activities. Of these, 10% no longer existed by the time we asked program 
coordinators to describe the programs several months later.Of the sampled activities for 
which we expected a response, we received only 3,691 (or 52%) completed coordinator 
surveys. 

The data on quality, then, come from approximately 3,700 reports completed by 
program coordinators from about 550 schools on fourteen of the different types of 
program activities. 

We developed 8 different indicators of program quality. These can be grouped into 
the three major categories shown in Figure 3. Most of these indicators are self- 
explanatory, but the “best practices — content and methods” score require explanation. 

Each of the activity questionnaires began with a question asking about the specific 
content of the activity. For prevention curriculum, for example, the booklet has a list of 
several topics that might be covered in the curriculum, ranging from religious teachings 
to self-esteem. This list contained a number of topics that had been shown in previous 
research to be related to problem behavior and had been included in successful 
prevention curricula. Figure 4 shows these topics. We scored each program activity as 
to the percentage of these desirable topics that were claimed by the program 
coordinator. We scored “best practices - content” in a similar fashion for all program 
categories for which prior research provided some guidance about what content had 
been shown to be effective. 

We used the same strategy to score the methods used in the program or activity. 
Figure 5 shows the methods that were keyed as desirable for behavior management 
programs. 
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Our report shows how each of these dimensions was measured and shows scores 
on each of these 8 dimensions of quality, by school level and location, and program 
type. We needed some way to boil all of this down to a bottom line. So, for each quality 
indicator for each program, we rated whether or not the level of implementation was 
“adequate.” We selected a cut-point for this rating such that an adequate rating 
denoted a level of implementation that might be expected to register effects on problem 
behavior - or at least we had some evidence that it had in the past. Figure 6 shows 
some examples of the cut-points we used for different types of prevention. We 
selected these cut-points by examining studies of programs in each category that had 
shown some positive effect on a measure of problem behavior, and setting the cut-point 
to be consistent with those studies. So, for example, we knew LST (Botvin et al., 1990) 
had registered effects on substance use and it had 16 sessions, so we selected 16 as 
the cut-point for the number of sessions for the prevention curriculum activity program 
type. And, although there have been only a handful of studies of mentoring, the Big 
Brother/Big Sister evaluation showed some positive effects and involved weekly contact 
between the mentor and mentee, so we chose 52 as the number of sessions below 
which the program would be considered “not adequate.” 

Figure 7 shows the percentage of attributes judged “adequate” for each activity 
type. We found that the quality of prevention programs in schools is low. For all types 
of programs, the mean proportion of adequacy criteria met is only .57. This means that 
for the average activity, only 57% of the indicators of quality were judged to be 
sufficiently strong to be expected to lead to a measurable difference in the desired 
outcomes. The summary index ranges from a low of .42 for services or programs to 
family members to .73 for security and surveillance activities. 

These results suggest that we need to attend to raising the quality of 
implementation of prevention programs in general, but they do not convey a sense for 
the large amount of variability in program implementation from activity to activity, even 
among activities of the same type. This is perhaps the most important finding from our 
national study. The percentage of variance in one of the quality indicators - the level of 
use, for example, that lies between program categories is only 5%. This means that 
most of the variability in quality is within program category. Another, perhaps more 
intuitive way to convey this point is to show examples of specific programs of the same 
type which differ in their quality of implementation. Figures 8 and 9 show what similar 
programs look like when implemented in two different schools. The first is a school 
planning intervention. You can see that in School A, school personnel are engaged in 
this activity only from time to time, while in school B they are engaged on a regular 
basis. In school A, the activity lasts a week, in school B, at least a year. In school A, 
students and staff participate one or twice a year, in School B, daily. And so on. The 
next overhead shows the characteristics of two similar behavior management 
interventions, both of which target high risk youths using behavioral strategies. But in 
school C, the activity lasts a month, and school D it continues for more than a school 
year. In school C, students participate monthly or less, while in school D, they 
participate more than once a day. In school C, none of the research-based methods are 
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used, and in school D, 88% are. Both of these types of prevention are effective 
according to the research summaries (Gottfredson, 1997; Gottfredson, Wilson, and 
Najaka, forthcoming), but it is likely that only the more strongly implemented model 
really works. 

Finally, we tested our hypotheses about the predictors of strength of program or 
activity implementation at the school level by reporting the aggregate-level correlations 
between characteristics of schools and schools’ prevention activities and the average 
quality of implementation in those schools. This activity generated hundreds of 
correlations, which are summarized in great detail in the full report. Figure 10 shows 
the bottom line from these analyses: A small number of predictors of the quality of 
prevention activities in schools predicted several of our indicators of program quality 
and quantity. These include the amount and quality of training, supervision, principal 
support for prevention activities, structure, the use of multiple sources of information 
(including district or other experts) in selecting activities to implement, integration of 
prevention as part of the regular school program, and local responsibility for initiating 
the activity. 

Conclusion 

Implementation varies a great deal from activity to activity. We see a wide range of 
implementation scores for each type of activity and on each dimension of quality, but 
the typical program receives a grade of “C” or better only on 57% of all of its quality 
dimensions. So there is plenty of room for improvement in the quality of 
implementation, and most likely the effectiveness of prevention activities can be 
boosted if we can find ways to improve the quality of what schools are already doing. 

There is every reason to expect that improving training, supervision, structure, and 
the availability of information can broadly and substantially improve the quality of 
school-based prevention of problem behavior. The present results also suggest that 
prevention interventions are most likely to be well implemented - and therefore have 
greater prospect of effectiveness - if they are integrated with the regular school 
program and initiated by school insiders. 
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Figure 1 

The Classification of Prevention Activity 



1 . Prevention curriculum, instruction, or training 

2. Behavioral or behavior modification interventions 

3. Counseling/social work/psychological/therapeutic interventions 

4. Individual attention/mentoring/tutoring/coaching 

5. Recreational, enrichment and leisure activities 

6. Referral to other agencies or for other services 

7. Improved instructional methods or practices 

8. Improved classroom management methods or practices 

9. Distinctive culture or climate for interpersonal exchanges - or 
improvements to intergroup relations or interaction between school & 
community 

10. Use of external personnel resources in classrooms 

1 1 . Youth roles in regulating and responding to student conduct 

12. School planning structure or process - or management of change 

13. Security and surveillance 

1 4. Services to families 

15. Rules, policies, regulations, laws, or enforcement 

16. Provision of information 

17. Reorganization of grades, classes, or school schedules 

1 8. Exclusion of weapons or contraband 

1 9. Alter school composition 

20. Training or staff development intervention 

21 . Architectural features of the school 

22. Treatment or prevention interventions for administration, faculty, or 
staff 
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Figure 2 

Hypothesized Factors Leading to Successful Program 

Implementation 

The following foster successful implementation of prevention programs: 

★ Organizational capacity (morale, history of few failed programs, 
staffing stability) 

★ Leadership and staff traits, past accomplishments 

★ Budget and support 

★ Organizational support (training, supervision, principal support) 

★ Program structure — manuals, implementation standards, quality 
control mechanisms 

★ Integration with normal school operations, local initiation, local 
planning, local information use 

★ Program feasibility — match between program design features and 
regular activities of the implementing school 

★ Little disorder 



“For whoever has, to him more will be given, and he will have 
abundance; but whoever does not have, even what he has will be 
taken away from him.” Matthew 13:12 
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Figure 3 

Measuring Program Quality - Three Groups of 

Indicators 



Technical Quality: 

★ % “best practices” used: content of activity 

★ % “best practices” used: methods 

★ Intensity- Number of lessons/sessions, duration, 
and frequency of student participation 

Extensiveness of Application: 

★ Level of use by school personnel 

★ Frequency of staff participation 

★ Frequency of program operation 

Extensiveness of Student Exposure: 

★ Ratio of program providers to students in the school 

★ Proportion of students exposed or participating 



Figure 4 

Measuring Best Practices ( Content ) - Prevention 
Curriculum, Instruction or Training 

Topics Covered 

• Social influence (recognizing & resisting, refusal skills) 

• Social problem solving skills (identifying problems, 
generating alternatives, etc.) 

• Self-management (goal setting, self-monitoring, self- 
reinforcement) 

• Attribution training 

• Communication skills (interpreting and processing social 
cues, nonverbal communication, negotiating) 

• Emotional control 

• Emotional perspective taking 

Instructional Strategies 

• Behavioral modeling 

• Role playing 

• Rehearsal and practice of skills 

• Use of cues to remind individual to display a behavior 
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Figure 5 

Measuring Best Practices ( Methods ) - Behavioral 
Programming or Behavior Modification 

Different specific behavioral or educational goals for 
different individuals or groups 

Always involves a method of monitoring or tracking 
behavior 

Always tracks behavior for a period of time before 
attempting to change it 

Always has specific written behavioral goals 

Always makes specific rewards or punishments in response 
to specific behaviors part of a written behavioral plan 

Tracks and responds to behavior daily or more often 

If student behavior does not change, different reinforcers or 
a different schedule are sought 

When desired behavior change occurs, rewards are faded 
(given less frequently) or made more difficult to earn 
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Figure 6 

How Was the “Adequacy” of Programs or Activities 

Judged? 

Example: Prevention Curriculum, Instruction or Training 



□ 


One or more persons conducting on a regular basis 


□ 


70% or more of content “best practices” 


used 


□ 


70% or more of method “best practices” 


used 


□ 


Contains 16 or more lessons 




□ 


Duration is longer than 1 month 




□ 


Occurs at least once weekly 





Example: Behavioral Programming or Behavior Modification 

□ One or more persons conducting on a regular basis 

□ 70% or more of content “best practices” used 

□ 70% or more of method “best practices” used 

□ Students participate at least daily 

Example: Mentoring 

□ One or more persons conducting on a regular basis 

□ 70% or more of content “best practices” used 

□ 70% or more of method “best practices” used 

□ 52 or more “sessions” 

□ Duration is at least one school year 

□ Sessions are at least weekly 
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Figure 7 

A Score Card on Prevention Activities in Schools: 
Percentage of Attributes Judged “Adequate” 

Activity type % 

Curriculum, Instruction, or Training 57 

Behavior Programming/Modification 47 

Counseling, Social Work, Psychological 45 

Mentoring/Tutoring/Coaching 57 

Recreation, Enrichment or Leisure 51 

Improved Instructional Practices 59 

Classroom Organization or Management 71 

Change Expectations for Behavior 64 

Intergroup Relations/School-Community 56 

School Planning Structure/Change Process 71 

Security & Surveillance 73 

Services for Family Members 42 

Use of External Personnel in Classroom 51 

Youth Participation in School Discipline 69 



Figure 8 

High and Low Quality School Planning Interventions 



Attribute 


School A 


School B 


Level of Use by 

School 

Personnel 


One or more person 
participating from 
time to time 


One or more person is 
conducting activity on 
a regular basis 


Duration 


One week 


At least a full school 
year 


Frequency of 
Participation - 
Students and 
Staff 


Once or twice per 
school year 


Daily 


Responsibility 
for Activity 


Principal and 
counselor 


Broad range of school 
staff, police, and 
community members 


Training 


Short demo. 


2-3 days of training 


Role & 

Accountability 


Participation is 
voluntary; 
participants not 
held accountable 


Required; participants 
are held accountable 
for the activity 
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Figure 9 

High and Low Quality Behavior Modification 



Interventions 


Attribute 


School C 


School D 


Level of Use by 

School 

Personnel 


One or more person 
has been trained 


One or more person is 
conducting activity on 
a regular basis 


Duration 


One month 


> One school year 


Frequency of 
Participation - 
Students 


Monthly or less 


More than once per 
day 


Percentage of 

Students 

Participating 


8% 


3% 


Best Practices: 
Content 


43% 


100% 


Best Practices: 
Methods 


0% 


88% 



Figure 10 

What Are the Most Important Predictors of Quality 
and Extensiveness of Prevention Activity? 

/ Extensiveness and quality of training 

/ Supervision of the activity 

/ Principal support for the activity 
/ The degree of structure or scriptedness of the activity 
/ Local responsibility for initiating the activity 
/ Use of multiple sources of information, including “experts” 
/ Activity is a part of the regular school program 
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